HyperMask: Talking Head Projected onto Real Objects

HyperMask is a system which projects an animated face onto a physical mask, worn by an actor. As the mask moves within a prescribed area, its position and orientation are detected by a camera, and the projected image changes with respect to the viewpoint of the audience. The lips of the projected face are automatically synthesized in real time with the voice of the actor, who also controls the facial expressions. As a theatrical tool, HyperMask enables a new style of storytelling. As a prototype system, we propose to put a self-contained HyperMask system in a trolley (disguised as a linen cart), so that it projects onto the mask worn by the actor pushing the trolley. Keyword: Talking Face, Mask Tracking, Facial Expression Synthesis, Lip Synchronization, Interactive Entertainment, HYPERMASK TALKING HEAD PROJECTED ONTO REAL OBJECT SHIGEO MORISHIMA AND TATSUO YOTSUKURA Faculty of Engineering, Seikei University 3-3-1 Kichijoji-Kitamachi, Musashino, Tokyo 180-8633 JAPAN E-mail: {shigeo, yotsu}@ee.seikei.ac.jp KIM BINSTED AND FRANK NIELSEN Sony Computer Science Laboratories 3-14-13 Higashi-Gotanda Shinagawa-ku Tokyo 141 E-mail: {kimbm, nielsen}@csl.sony.co.jp CLAUDIO PINHANEZ IBM T.J. Watson Research 30 Saw Mill River Rd. (Route 9A) Hawthorne, NY 10532 E-mail: [email protected] HYPERMASK is a system which projects an animated face onto a physical mask, worn by an actor. As the mask moves within a prescribed area, its position and orientation are detected by a camera, and the projected image changes with respect to the viewpoint of the audience. The lips of the projected face are automatically synthesized in real time with the voice of the actor, who also controls the facial expressions. As a theatrical tool, HYPERMASK enables a new style of storytelling. As a prototype system, we propose to put a self-contained HYPERMASK system in a trolley (disguised as a linen cart), so that it projects onto the mask worn by the actor pushing the trolley.